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THE  PROBLEM 

Background 

The  overall  mission  of  the  Academic  instructor  Course,  Academic  Instructor  and 
Allied  Officer  School  is  to  improve  the  teaching  skills  of  selected  Air  Force  in¬ 
structors  in  order  for  them  to  perform  more  effectively  in  teaching  assignments. 
AIC  does  not  impose  restrictions  on  the  students  who  attend  the  course  or  require 
prerequisites  but  leaves  the  task  of  selection  to  the  organizations  who  will  employ 
the  talents  of  the  instructors  after  completing  the  course.  Since  the  clientele  that 
composes  the  normal  classes  is  so  heterogeneous,  a  basic  fundamental  core  of  cur¬ 
riculum  hours  is  presented  to  all  students.  In  order  to  gain  information  concerning 
the  student’s  knowledge  of  the  subjects  to  be  presented  and  to  provide  guidance  for 
faculty  working  with  the  students,  a  pre-course  test  is  administered  prior  to  student 
exposure  to  the  curriculum.  However,  in  an  effort  to  provide  maximum  individ¬ 
ualized  instruction  for  the  students,  a  more  accurate  assessment  of  the  students’ 
state  of  knowledge  must  be  made.  The  students’  knowledge  and  level  of  proficiency 
in  the  subject  matter  areas  must  be  determined.  There  appear  to  be  two  basic 
reasons  for  investigating  methods  and  techniques  of  measuring  these  “states  of 
knowledge.”  These  are:  ( 1 )  the  need  to  more  accurately  assess  the  students’  knowl¬ 
edge  of  the  curriculum  upon  entry  into  the  course  in  order  to  provide  maximum  in¬ 
dividualization  of  the  curriculum,  and  (2)  the  need  for  more  precise  data  upon 
which  to  base  curriculum  decisions. 
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Student  Differences 


Currently,  faculty  members  at  AIC  strive  to  meet  the  individual  differences  of  stu¬ 
dents  and  to  emphasize  the  need  for  individualized  instruction  with  their  students, 
who  will  become  instructors.  Most  faculty  advisors  rely  heavily  upon  observations 
of  student  performance,  knowledge  of  the  AIC  curriculum,  and  experiences  with 
the  students  to  guide  their  instructional  actions.  Additionally,  faculty  advisors  are 
provided  data  on  each  student  from  a  reading  skill  test  (Nelson-Denny )  and  a  pre¬ 
course  examination  that  is  a  representative  sample  of  the  AIC  curriculum  hours. 
However,  the  data  are  put  to  limited  use  since  there  is  some  question  as  to  their 
accuracy  and  dependability.  A  more  valid  and  reliable  approach  to  pre-course 
testing  should  make  it  possible  for  the  faculty  to  better  assist  the  student  based  up¬ 
on  his  actual  needs. 

More  Precise  Information  about  State  of  Knowledge 

Although  the  current  course  pretest  yields  some  information  for  the  faculty  ad¬ 
visors  use  in  individualizing  instruction,  there  is  a  need  for  a  test  that  will  yield  ad¬ 
ditional  precise  information  as  to  the  students'  actual  levels  of  achievement.  The 
current  test  does  not  employ  a  correction  for  guessing  formula  and.  in  fact,  en¬ 
courages  the  student  to  respond  to  all  items  on  the  exam,  since  the  faculty  advisors 
need  information  on  all  areas  of  the  curriculum.  If  the  lest  results  gave  a  more  accu¬ 
rate  assessment  of  the  students’  actual  state  of  knowledge  (level  of  proficiency)  the 
advisor  could  better  tailor  his  teaching  strategy  in  small  group  (seminar)  activities  to 
meet  the  actual  needs.  Although  this  report  may  not  be  conclusive  in  all  aspects  of 
its  exploration,  the  following  are  a  number  of  needs  which  the  research  at  AIC  can 
serve: 

1.  The  need  for  a  more  accurate,  concrete  assessment  of  the  students'  states  of 
knowledge.  This  would  be  in  harmony  with  Alt  ’s  concept  ol  meeting  the  students' 
needs  through  individualized  instruction. 

2.  The  need  for  the  faculty  to  better  meet  the  individual  needs  ol  students  through 
proper  guidance  and  appropriate  teaching  and  reteaching  strategics. 

3.  The  need  for  more  accurate  and  meaningful  grades  on  all  written  tests  and  more 
accurate  data  for  computing  the  final  class  standings. 

Hypotheses 

The  overall  problem  in  this  study  is  to  determine  if  Valid  Confidence  Testing 
techniques  can  provide  a  more  accurate  assessment  of  the  students'  knowledge  of 
the  subject  matter  and  yield  more  useful  information  for  the  faculty  than  standard 


choice  testing.  The  broad  hypothesis  of  the  study  is  that  such  testing  techniques 
can  yield  such  information  and  serve  as  a  useful  tool  in  predicting  a  student’s  suc¬ 
cess  in  the  course. 


RESEARCH  PROCEDURES 

The  Nature  of  the  Data 

Two  separate  analyses  were  conducted  in  order  to  include  a  wide  range  of  student 
backgrounds  and  experiences  and  to  allow  comparisons  of  varying  student  groups. 
Class  69-A  was  the  first  to  be  analyzed.  It  consisted  primarily  of  officers  designated 
for  either  instructor  or  instructor-supervisor  duty  throughout  the  Air  Force.  The 
educational  level  of  Class  69-A  averaged  slightly  higher  than  the  baccalaureate 
level.  The  second  class  to  be  analyzed  was  Class  69-B.  This  class  consisted  primarily 
of  NCO’s  and  a  few  civilians,  most  of  whom  were  employed  as  technical  instructors. 
Air  Training  Command  provided  the  largest  single  group  of  students  in  that  class. 
Education  varied  more  widely  in  this  class  than  in  69-A,  with  a  range  of  8th  grade 
to  the  B.A.  level.  Average  educational  level  for  Class  69-B  was  1 2.9  years. 

The  analysis  groups  consisted  of  68  and  83  students,  respectively.  These  numbers 
represented  approximately  one-half  of  the  students  in  each  class  and  were  randomly 
selected  by  seminar. 

Twelve  vaiiables  were  collected  on  each  individual  for  the  analysis  and  were  pro¬ 
vided  by  the  records  maintained  by  the  Department  of  Educational  Evaluation  of 
the  Academic  Instructor  and  Allied  Officer  School. 

Statistical  Analysis 

This  analysis  was  conducted  on  the  GE  635/645  Time-Sharing  System  using  a  re¬ 
mote  terminal  located  in  the  Information  Sciences  Directorate  of  the  Air  Univer¬ 
sity  Institute  for  Professional  Development.  Data  were  input  manually  through  the 
console  and  analyzed  with  both  “canned"  library  programs  and  a  special  purpose 
program  designed  and  coded  by  a  member  qf  the  Department  of  Instructional  Tech¬ 
nology,  AIAOS.  The  core  of  the  analysis  consisted  of  a  12  x  12  matrix  of  Pearson 
product  moment  correlations  for  each  group. 

Materials  Used 

Both  classes  used  the  materials  and  techniques  developed  by  The  Shuford-Massengill 
Corporation  for  taking  confidence  tests.  These  materials  consisted  of  ( I )  SCoRule 
response  aids,  (2)  answer  sheets  designed  for  use  with  the  SCoRule,  and  (3)  scoring 


tables  for  conversion  of  levels  of  confidence  into  a  numerical  score.  In  addition  to 
these  materials,  the  students  were  provided  with  the  standard  multiple-choice  tests 
administered  to  all  AIC  classes  and  the  standard  answer  sheet  that  is  used  by  Air 
University.  All  students  were  given  instructions  in  the  use  of  the  materials  via  video 
tape  and  were  administered  a  short,  four-item  practice  test  prior  to  the  actual 
testing. 

THE  RESULTS 


This  section  of  the  report  will  describe  the  major  findings  of  the  study.  Because  of 
the  large  number  of  relationships  that  exist  in  a  12  x  12  matrix  many  must  be 
omitted;  only  the  most  important  relationships  will  be  discussed.  If  other  relation¬ 
ships  are  of  interest,  they  can  be  found  in  Appendix  A. 

Specific  Hypotheses 

Predictive  Validity 

Hypothesis:  Confidence  tests  will  provide  a  more  valid  prediction  of  the  students' 
success  in  the  course  and  in  practice  teaching  lessons  than  will  choice  tests. 

The  reasoning  behind  this  hypothesis  is  that  a  student  who  can  accurately  evaluate 
his  own  knowledge  and  is  confident  in  his  knowledge  of  the  subject  matter  will  per¬ 
form  at  a  high  level  in  the  practice  teaching  lessons  (PT's)  which  are  the  core  of  the 
curriculum's  graded  activities.  The  results,  listed  in  detail  in  Tables  1 4,  however,  do 
not  bear  this  out.  For  Class  69-A,  there  was  no  statistically  significant  correlation 
between  confidence  pretest  scores  and  practice  teaching  efforts.  For  the  enlisted 
class  (69-B),  the  correlation  was  even  lower.  The  officer  class  (69-A)  showed  a  slight 
positive  correlation  between  pretest  confidence  scores  and  five  depc'ndent  variables: 
Teaching  Interview  PT  (.15),  Guided  Discussion  PT  (.25),  Lecture  PT  (.34),  Final 
PT  (.31),  and  Final  Rank  Order  (.45).  The  pretest  scores  of  the  enlisted  class 
showed  the  following:  Demonstration  Performance  PT  (24)  .Guided  Discussion 
PT  (.28),  Lecture  PT  (.21),  Final  PT  (.21),  and  Final  Rank  Order  (.38).  It  should 
be  noted  that  the  choice  pretest  score  for  69-A  correlated  with  the  same  variables  as 
follows:  Teaching  Interview  (.17),  Guided  Discussion  (.21 ),  Lecture  PT  (.32).  Final 
PT  (.17),  and  Final  Rank  Order  (.42).  The  enlisted  class  had  the  following  corre¬ 
lations  between  choice  pretest  scores  and  output  variables:  Demonstration  Per¬ 
formance  PT  (.33),  Guided  Discussion  PT  (.25),  Final  PT  (.35),  Lecture  PT  (.25) 
and  Final  Rank  Order  (.22).  Except  for  the  Guided  Discussion  PT.  all  correlations 
for  this  class  between  choice  test  scores  and  PT’s  were  slightly  higher  than  those  of 

*  Demonstration  Performance  Method  is  substituted  for  the  Teaching  Interview  Method  with 
enlisted  Classes.  Difficulty  is  comparable  with  the  Teaching  interview. 


Table  1.  Correlations  between  Pretest  (Choice  and  Confidence)  and 
Final  Rank  Order.  Class  69B  is  in  medium  type. 


confidence  scores  and  PT’s.  We  can  infer  then,  that  for  officer  classes  the 
hypothesis  is  only  partially  confirmed,  but  not  confirmed  for  enlisted  classes.  Con¬ 
fidence  pretest  scores  have  a  slight  positive  correlation  with  practice  teaching 
lessons  and  final  rank  order,  but  not  significantly2  better  than  choice  scores  for 
officer  classes.  For  enlisted  classes,  choice  scores  are  better  predictors  than  con¬ 
fidence  scores,  but  are  at  the  best  modest. 


Although  normal  sampling  theory  doer  not  apply  in  this  study,  a  test  of  statistical  significance 
is  applied  at  a  general  indication  of  the  importance  of  the  relationships.  A  coefficient  of  roughly 
±  .20  would  be  significant  at  the  .01  level  of  samples  of  this  size,  indicating  that  there  it  only 
one  chance  in  a  hundred  that  the  observed  correlation  it  attributable  to  sampling  error  rather 
than  to  a  real  relationship  of  the  data.  This  criterion  is  borrowed  from  sampling  theory  at  a 
meant  of  establishing  an  arbitrary  cutoff  point  for  evaluating  coefficients. 

In  many  cases  the  correlations  may  be  '‘significant”  in  the  statistical  tense,  but  not  very  mean¬ 
ingful  to  the  member  or  policy  maker.  Therefore,  the  reader  may  wish  to  establish  hit  own 
criterion  of  meaningfulness.  In  the  verbal  descriptions  of  this  report,  correlations  leu  than 
i  .40  are  not  considered  very  meaningful  unleu  part  of  a  particular  suggestive  pattern. 

The  formulas  used  for  establishing  significance  levels  appear  in  Helen  M.  Walker  4  Joseph  Lev, 
Stitiitictl inference  (New  York:  Holt,  Rinehart  4  Winston,  1953)  p.  251. 


Table  3.  Relationships  of  Reading  Skill  and  Practice  Teaching  Lessons. 
Class  69B  is  in  medium  type. 


Aw— ament  of  Studtnt  Knowledge 

Hypothesis:  The  scores  attained  on  a  confidence  test  will  he  a  more  accurate  assess¬ 
ment  of  student  knowledge  than  will  choice  scores. 

The  baiit  for  'his  hypotheiii  it  that  the  scores  attained  by  students  taking  a  con¬ 
fidence  test  will  be  a  more  accurate  reflection  of  their  actual  level  of  knowledge  be¬ 
cause  they  will  more  honestly  and  realistically  analyze  the  items  and  their  own 
knowledge  before  responding.  The  scoring  system  of  confidence  testing  makes  it 
in  the  best  interest  of  the  student  not  to  guess  on  items  of  which  he  is  not  sure  but 
rather  to  split  his  confidence  among  the  answers  he  considers  possible.  The  more 
confidence  he  has  in  (he  correct  answer,  the  more  points  he  receives.  Also,  he  is  not 
penalized  if  he  doesn't  know  the  answer  to  an  item  with  which  he  is  not  familiar. 
He  may  split  his  confidence  equally  among  all  the  possible  answers,  thus  receiving 
some  credit  for  his  “honesty"  rather  than  be  placed  in  a  situation  in  which  he 
must  guess.  This  also  eliminates  the  situation  in  which  he  is  penalized  for  guessing 
incorrectly  when  a  correction  for  guessing  formula  is  used.  Theoretically,  then,  it  is 
in  his  best  interest  not  to  guess  on  any  of  the  items  on  the  test  Therefore,  the  re¬ 
sults  should  be  a  more  accurate  reflection  of  his  actual  knowledge. 

The  graphs  listed  in  Tables  5  and  6  display  the  degree  to  which  the  students  in  both 
classes  realistically  evaluated  information.  This  is  based  upon  the  percentage  of 
times  the  students  displayed  complete  confidence  and  were  correct  (Z)  and  the 
percentage  of  times  they  displayed  complete  confidence  in  an  answer  and 
were  incorrect  (A).  When  usingthe  SCoKule,  the  letter  "l"  represents  complete  con¬ 
fidence  in  an  alternative  while  "A"  indicates  that  there  is  no  possibility  that  the  al¬ 
ternative  to  which  it  is  assigned  can  be  the  correct  answer. 

The  more  accurately  the  students  evaluated  information,  the  closer  the  lines 
should  approach  the  identity  line  (which  is  an  indication  of  ideal  realism  when 
assigning  confidence  to  alternatives).  It  should  be  noted  that  with  the  exception 
of  the  post  test  for  Class  69-B,  each  time  the  students  were  administered  a  test, 
the  more  nearly  the  results  approached  the  identity  line.  With  more  tests, 
administered  over  a  longer  period  of  time,  it  is  reasonable  to  assume  that  the  stu¬ 
dents  would  become  more  proficient  in  the  evaluation  of  their  own  knowledge. 
The  post  test  response  validity  line  for  Class  69-B  dropped  from  the  pre¬ 
vious  test  that  was  administered.  This  may  be'the  result  of  theii  belief  that  the  choice 
test  results  and  not  the  confidence  scores  were  to  be  counted  into  their  final  rank 
order  standings.  Thus,  they  may  have  resorted  to  the  normal  procedure  of  re¬ 
sponding  to  most  items  with  complete  confidence  although  they  lacked  the  knowl¬ 
edge  to  justify  such. 

It  should  be  noted  that  the  officer  class  (69-A)  consistently  evaluated  information 
more  realistically.  Each  test  that  was  administered  achieved  a  higher  level  of  re¬ 
sponse  validity  than  did  those  of  the  enlisted  class.  This  would  tend  to  indicate  that 


Table  5.  Response  Validity  for  AIC  (’lass  Pl)A  (  Ml  Students). 
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■  Pretest 

■  Unit  Survey  No.  I 

*  Unit  Survey  No.  2 

*  Post  Test 


Table  6.  Response  Validity  for  AIC  Class  69B  (All  Students). 
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(he  officer  classes  were  better  able  to  evaluate  information  than  enlisted  classes, 
perhaps  because  of  their  higher  level  of  formal  education  and  tiainmg. 

Another  measure  of  the  accuracy  of  the  assessment  of  students'  knowledge  is  the 
degree  to  which  confidence  scores  deviate  from  their  choice  scores.  Since  the  stu¬ 
dents  are  receiving  credit  for  partial  knowledge,  the  scores  of  the  confidence  tests 
should  be  consistently  higher  than  those  they  would  receive  it  they  were  taking 
the  test  in  the  standard  choice  mode. 

Tables  7  and  8  indicate  the  scores  for  the  pretest  and  post  test  for  Class  The 
confidence  scores  are  plotted  on  the  vertical  axis,  while  the  choice  scores  are  on  the 
horizontal  axis.  On  the  pretest,  all  but  five  students  scored  higher  on  the  confidence 
test  than  they  did  when  taking  it  as  a  choice  test.  Likewise.  59  other  students 
achieved  higher  choice  scores  than  confidence  scores  on  the  post  test.  These  stu¬ 
dents  were  able  to  guess  successfully  when  faced  with  an  all-or-nothing  situation. 
However,  it  should  be  noted  that  the  largest  deviation  from  confidence  score 
was  only  2.5  points  and  this  occurred  only  with  one  student.  The  mean  deviation 
of  the  5-choice  scores  was  only  .675  higher  than  the  confidence  scores.  This  dif¬ 
ference  would  appear  to  he  insignificant,  since  the  majority  (6.1)  of  the  confidence 
scores  were  much  higher  than  those  of  choice. 

The  final  index  of  the  accuracy  of  measuring  student  knowledge  reported  in  this 
study  is  reflected  in  Appendix  Band  C.  These  tables  contain  the  comparison  be¬ 
tween  choice,  confidence  scores  and  rank  orders  for  each  student  on  the  post  test 
for  both  Classes  69-A  and  69-B.  The  officer  class  is  characterized  by  many  rank 
order  inversions  occurring  throughout  the  entire  distribution.  As  might  be  expected, 
there  were  generally  minimal  changes  in  position  at  the  high  and  low  ends  of  the 
rank  order,  with  most  significant  changes  occurring  in  the  middle  lW'f  of  (he  cases. 
The  most  significant  change  in  rank  order  occurred  with  student  1-1.  His  position 
changed  from  twenty-fifth  in  the  group  when  calculated  by  his  choice  score,  to 
eighth  with  his  confidence  score.  His  raw  score  differed  from  65  (choice)  to  68.60 
(confidence)  5.90  points.  This  difference  was  the  credit  he  received  for  partial 
knowledge  with  the  confidence  test,  that  is,  he  actually  had  more  knowledge  than 
his  choice  score  indicated.  The  other  extreme  is  illustrated  by  students  H  -  2. 
K  •  5,  B  •  6,  and  H  •  3.  In  each  of  these  cases  there  was  a  significant  drop  in  their 
rank  order  standing  from  that  indicated  by  their  choice  scores.  Students  H  -  2, 
K  -  5  and  B  -  6  all  achieved  a  raw  score  of  58  and  were  ranked  forty-first  when 
taking  the  test  in  the  choice  mode.  This  would  indicate  they  all  had  equal  knowl¬ 
edge  and  were  ranked  the  same.  However,  the  confidence  scores  were  59.00.  58.44 
and  58.37  respectively,  and  they  were  ranked  51st,  52nd  and  53rd,  thus  more 
accurately  reflecting  the  differences  that  did  exist.  This  differentiation  resulted 
from  the  credit  given  by  confidence  testing  for  partial  knowledge.  One  student, 
F  -  2,  had  a  raw  score  difference  of  2.07  points,  and  was  the  only  student  that  did 
not  have  some  change  in  his  rank  order  position. 


Table  7.  Comparison  of  Pretest  Confidence  and  Choice  Scores,  Class 
69A.  Confidence  raw  scores  are  plotted  on  the  vertical  axis  while  the 
choice  raw  scores  are  on  the  horizontal  axis. 
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Table  8.  Comparison  of  Post  Test  Confidence  and  Choice  Scores,  Class 
69A.  Confidence  raw  scores  are  plotted  on  the  vertical  axis  while  the 
choice  raw  scores  are  on  the  horizontal  axis. 


With  the  enlisted  class,  69-B,  the  reversals  in  position  were  even  more  pronounced. 
The  greatest  change  in  rank  order  position  occurred  with  student  T  •  5.  His  position 
changed  from  60.5/84  (choice)  to  22/84  (confidence),  a  difference  of  38.5  po¬ 
sitions.  The  credit  for  his  partial  knowledge  gave  him  a  raw  see:?  of  38.39  (confi¬ 
dence)  as  compared  to  his  choice  score  of  29-a  difference  of  9.39  points.  There 
are  many  other  similar  cases  in  69-B  with  large  reversals  of  position.  The  other  ex¬ 
treme  is  illustrated  with  Student  H  -  8.  He  was  ranked  49/84  based  upon  his  confi¬ 
dence  raw  score  of  36.20,  but  his  choice  score  of  43  ranked  him  4/84.  The  primary 
difference  was  his  success  at  guessing  the  correct  answer  when  taking  the  test  in  the 
choice  mode.  Two  students,  Q  -  2  and  N  •  5,  received  different  taw  scores,  but  failed 
to  change  positions. 

In  general,  the  data  presented  in  this  section  of  the  report  tends  to  confirm  the  hy¬ 
pothesis  that  a  more  accurate  assessment  of  the  students’  actual  state  of  knowledge 
can  be  attained  with  the  use  of  confidence  testing.  Most  of  the  students  in  both 
classes  would  have  been  mis-classified  in  terms  of  their  actual  achievement  had  only 
the  choice  scores  and  accompanying  rank  order  grades  been  considered,  since  some 
98'T  of  the  students  had  changes  in  rank  order  positions.  It  seems  likely  that  the 
more  exposure  the  students  have  to  confidence  tests,  the  more  realistically  they  will 
evaluate  test  items  and  their  own  knowledge,  thus  providing  a  more  valid  indication 
as  to  their  actual  level  of  achievement. 

Reliability 

Hypothesis:  Confidence  test  results  will  be  more  reliable  than  choice  tests  since 
there  is  a  significant  reduction  in  guessing. 

Perhaps  the  greatest  single  factor  that  affects  the  reliability  of  any  test  is  that  of 
guessing  on  the  part  of  the  students  taking  the  test.  This  factor,  according  to 
J.C.  Nunnally,  Victor  H.  Noll  and  other  authorities  in  evaluation,  along  with  the 
clarity  of  the  individual  items,  environmental  conditions  under  which  the  test  is  ad¬ 
ministered,  and  the  instructions  that  are  provided  the  students  determine,  in  large 
part,  how  reliable  the  instrument  will  be.  Since  all  of  the  above  mentioned  factors 
can  be  controlled  (with  the  exception  of  the  guessing),  for  either  a  choice  or  confi¬ 
dence  test,  it  is  reasonable  to  expect  that  the  test  that  least  encourages  the  students 
to  guess  would  be  the  more  reliable  instrument. 

Students  in  Gasses  69-A  and  69-B  were  administered  the  pre  and  post  tests  in  both 
choice  and  confidence  modes.  The  answer  sheets  from  the  choice  tests  were  graded 
with  an  optical  scanner,  and  scored  with  a  general  computer  scoring  program.  Each 
test  was  statistically  analyzed  by  the  computer  to  provide  print-outs  of  the  individ¬ 
ual  it'm  analysis  data  (Ease  and  Differentiation  Indices  as.  well  as  alternative  se¬ 
lections),  and  overall  test  data  which  included  the  Measures  of  Central  Tendency, 
Variability  and  the  Reliability  Coefficient  and  computed  with  Kuder-Richardson 


Formula  2 1.3  The  confidence  tests  were  manually  scored  and  all  reliability  co¬ 
efficients  computed  using  Kuder-Richardson  20.  In  using  K-R  20  to  compute  the 
reliability  coefficient,  the  numerical  values  of  the  levels  of  confidence  assigned  to 
the  correct  answer  were  used.  In  Table  9  the  significant  measures  of  the  test 
are  listed. 


Table  0.  Test  Statistics  for  69A  and  69B  Pro  and  Post  Tests. 


CLASS  69-A 


Pretest 


Post  Test 


Choice 

Confidence 

Choice 

Confidence 

Mean 

13.00 

17.09 

58.85 

60.96 

Median 

14.0 

17.84 

57.0 

62.49 

Mode 

13.0 

58 

66.00 

Range 

19.0 

14.49 

41.0 

35.30 

Standard  Dev. 

4.02 

3.4 

8.28 

10.3 

Reliability 

.64 

.70 

.82 

.90 

CLASS  69-B 

Pretest 

Post  Test 

Choice 

Confidence 

Choice 

Confidence 

Mean 

11.45 

14.68 

32.5 

35.46 

Median 

11 

14.90 

31 

35.39 

Mode 

12 

13.34 

31 

37.20/34.00/33.02 

Range 

20 

11.17 

25.0 

23.09 

Standard  Dev. 

3.12 

3.01 

5.92 

7.69 

Reliability 

.38 

.43 

.81 

.84 

1  l  or  ease  of  compulation,  since  it  had  to  be  performed  manually,  the  reliability  co¬ 
efficients  for  all  choice  tests  were  recomputed  using  Kudcr-Kichardson  formula  20.  Thus 
a  more  meaningful  comparison  can  be  made  with  the  confidence  test  scores. 


In  all  cases,  there  were  significant  increases  in  the  reliability  of  the  tests.  This  would 
indicate  that  there  is  some  validity  to  the  hypothesis.  In  addition  to  the  data  men¬ 
tioned  above,  we  can  by  inference  ascertain  that  there  should  be  a  substantial  in¬ 
crease  in  the  reliability  of  the  tests  when  they  are  administered  using  confidence 
techniques  since  there  is  a  significant  reduction  of  guessing.  We  can  make  this  in¬ 
ference  by  observing  the  data  presented  as  “States  of  Knowledge.”  The  states  of 
knowledge  mentioned  are  derived  from  the  level  of  confidence  placed  upon  the 
correct  answer  by  the  student.  If  he  has  placed  complete  confidence  on  the  correct 
answer,  he  is  said  to  be  “Well  Informed"  (W).  If  he  places  most  of  his  confidence, 
but  not  all  on  the  correct  answer,  he  is  classified  as  “Informed”  (I).  If  he  places 
equal  confidence  in  either  two  or  three  of  the  possible  answers,  he  is  “Partially  In¬ 
formed”  (P).  When  he  equally  splits  his  confidence  among  all  the  possible  an- 
swers  that  is,  he  has  no  ideas  as  to  which  answer  might  be  correct  he  is  “Unin¬ 
formed"  (U).  When  most  of  a  student’s  confidence,  but  not  all,  is  placed  on  an  in¬ 
correct  answer,  he  is  “Misinformed”  (M).  Finally,  when  he  has  complete  confidence 
in  an  incorrect  answer,  he  is  classified  as  being  “Completely  Misinformed”  (C). 
Thus  by  being  able  to  split  confidence  among  possible  answers,  we  tend  to  eliminate 
guessing. 

Table  10  provides  the  totals  of  the  categorized  states  of  knowledge  and  the 
percentages  of  students  within  each  category  for  the  pre  and  post  tests  for  each 
class.  From  these,  we  can  show  the  number  of  guessing  “situations”  which  occurred 
for  each  test  that  were  eliminated  by  using  the  confidence  testing  method. 

With  the  officer  class,  there  were  491  instances  in  which  guessing  was  eliminated  on 
the  pretest.  That  is.  the  students  split  their  confidence  equally  (uninformed)  274 
times  and  divided  among  either  two  or  three  alternatives  (partially  informed)  217 
times.  Thus,  there  were  491  instances  in  which  some  degree  of  guessing  occurred 
on  this  test  when  it  was  taken  in  the  choice  mode.  Similarly,  there  were  438 
situations  in  which  guessing  was  eliminated  on  the  post  test  for  this  class.  There¬ 
fore,  we  can  infer  that  the  pretest  for  Class  69-A,  when  taken  as  a  choice  test  was 
contaminated  by  guessing  24.2%  of  the  time.  On  the  post  test  this  occurred  8.3%  of 
the  time. 

The  enlisted  class  had  a  slightly  higher  guess  rate  on  the  choice  test  than  did  the 
officer  class.  On  the  post  test,  there  were  331  Uninformed  responses  and  29 1  Par¬ 
tially  Informed  responses  for  a  total  of  622  situations  in  which  guessing  occurred 
when  taken  as  a  choice  test.  This  indicates  that  on  the  pretest  for  69-B  students 
guessed  at  29.7%  of  the  possible  responses.  The  total  for  the  post  test  for  this 
class  was  499,  or  1 1 .9%. 

If  guessing  on  a  test  can  be  eliminated  or  at  least  minimized  as  indicated  above,  then 
it  is  likely  that  the  test  will  be  a  more  reliable  vehicle.  Thus,  as  indicated  in  the 
initial  portion  of  this  section,  and  confirmed  with  the  above  data,  this  hypothesis 
appears  to  be  valid. 


~*~n  X 


Information 

Hypothesis:  Confidence  tests  yield  more  information  that  can  he  used  in  policy 
making  for  a  multi-tracked  curriculum  than  do  choice  tests. 

A  capability  for  diagnosing  student  strengths  and  weaknesses  is  needed  if  those 
assigned  the  responsibility  of  making  curriculum  decisions  are  to  act  intelligently  to 
meet  the  individual  needs  of  the  students.  With  AIC  attempting  to  individualize 
instruction,  more  than  mere  intuition  is  needed  if  those  needs  are  to  be 
met.  There  are  only  two  basic  sources  of  empirical  data  available  at  the  present. 
These  being  the  pretest  and  the  Nelson-Denny  Reading  Test.  The  course  pretest  has 
been  discussed  earlier  in  this  report.  The  reading  test  attempts  to  measure  a  stu¬ 
dent’s  reading  rate,  vocabulary  and  comprehension.  As  indicated  in  this  report,  the 
course  pretest  has  only  minimal  predictive  validity,  and  when  administered  as  a 
choice  or  confidence  test,  yields  little  information  that  can  be  relied  upon  as  a  true 
indicator  of  how  well  the  student  will  perform  on  practice  teaching  lessons  or  how 
well  he  will  do  in  the  course.  Since  the  frequency  of  guessing  is  high  with  the  choice 
test,  too  many  inferences  have  to  be  made  as  to  the  actual  knowledge  of  the  subject 
matter.  Therefore,  little  realistic  course  planning  can  be  made  using  the  choice  pre¬ 
test  as  a  guide.  The  Nelson-Denny  Reading  Test  is  an  excellent  measure  of  a  stu¬ 
dent's  ability  to  read  and  comprehend  the  material,  but  a  previous  study  has  indi¬ 
cated  that  there  is  only  a  slight  correlation  between  reading  skills  (as  measured  with 
the  Nelson-Denny)  and  success  on  practice  teaching  lessons  in  the  course.4 

By  virtue  of  the  fact  that  confidence  testing  categorizes  students’  states  of  knowl¬ 
edge  into  six  basic  levels,  it  becomes  possible  to  determine  curriculum  hours  to  be 
presented,  and  the  type  of  teaching  strategy  that  should  be  employed.  Regarding 
curriculum  hours  which  need  to  be  presented.  Appendix  D  contains  the  results  of 
the  pretest  administered  as  a  confidence  test  to  Class  69- A.  The  table  provides  a 
breakdown  of  the  various  states  of  knowledge  by  item  and  the  percentage  of  stu¬ 
dents  in  that  classification.  The  left  column  indicates  the  objective  that  is  being 
measured.  The  first  seven  items  on  the  test  measure  hours  in  the  Educational 
Fundamentals  area;  items  8  through  14  measure  hours  in  the  Communicative 
Skills  area;  IS  through  19  cover  the  Educational  Methods  hours:  and  the  final  6 
items  measure  the  area  of  Educational  Evaluation. 


4  In  the  study,  An  Analysis  of  the  Relationship  Between  Student  Background  Characteristics 
and  Success  at  the  Academic  Instructor  Course,  Captain  Meredith  W.  Walts,  Jr.,  found  the 
correlations  between  reading  skills  and  final  rank  order  lo  be  .29  for  an  officer  class  and 
.43  for  an  enlisted  class.  Additionally,  the  correlations  between  reading  skill  and  practice 
teaching  lessons  for  officer  classes  to  be:  Teaching  Interview  PT  -  .16,  Guided  Discussion 
PT  •  .14,  Lecture  PT  -  .27.  and  final  PT  •  .08.  I'or  enlhled  classes:  Demonstraiion  Per¬ 
formance  PT  •  .19,  Guided  Discussion  PT  •  .23,  Lecture  PT  •  .28,  And  final  PT  •  .16. 


Table  10.  Summary  of  States  of  Knowledge. 


The  data  indicates  that  most  students  are  either  informed  or  well  informed  on  Items 
1,  2,  4  and  7.  This  we  would  expect,  since  most  of  the  officers  have  had  training 
and  experience  in  Psychology  (Items  I  &  2),  Counseling  (4),  and  Group  Dynamics 
(7).  It  is  also  noted  that  only  1 5%  are  informed  or  better  on  the  Educational 
Theories  (Item  3),  27%  in  Creative  Thinking  (5)  and  1 2r/>  on  Effective  Thinking  (6). 
Based  upon  the  large  percentages  of  students  who  were  informed  or  well  informed 
on  Items  1,  2,  4  and  7,  that  instruction  might  be  presented  only  to  those  students 
who  have  lower  states  of  knowledge  and  offered  as  supplementary  material  to  those 
who  already  are  knowledgeable.  It  can  also  be  noted  that  Items  4,  S,  6  and  7  also 
have  a  significant  number  of  students  who  are  either  misinformed  or  completely 
misinformed.  This  would  dictate  that  these  students  should  attend  instruction 
covering  this  material  and  that  the  strategy  used  to  teach  them  must  be  one  that 
will  consider  that  negative  transfer  of  learning  might  interfere  with  their  receptivity. 
This  is  inferred  from  the  large  percentage  of  students  who  placed  a  great  deal,  or  all, 
of  their  confidence  on  an  incorrect  alternative.  If  they  are  that  positive  that  their 
answer  is  correct,  then  it  may  be  difficult  for  them  to  receive  the  material  without 
interference  from  previously  learned  material. 

In  the  Communicative  Skills  area  (Items  8  through  14).  it  is  noted  that  more  than 
half  the  group  is  either  informed  or  well  informed  on  each  area  measured.  This  in¬ 
dicates  that  it  might  be  well  to  consider  presenting  the  material  on  an  optional  basis 
for  the  informed  students  and  as  individualized  remedial  instruction  for  the  re¬ 
mainder  of  the  group.  It  appears  that  a  sufficient  number  of  the  students  are  knowl¬ 
edgeable  enough  to  prohibit  the  presentation  of  a  “standard"  package  to 
all  students. 

In  the  Education  Methods  portion  of  the  test  (Items  IS  through  19)  the  levels  of 
knowledge  are  fairly  well  distributed  among  the  six  states  of  knowledge.  However, 
there  is  a  definite  tendency  for  more  students  to  be  misinformed  about  methods 
and  lesson  planning  than  in  the  preceding  area.  Teaching  strategy  must  be  altered  to 
compensate  for  the  negative  transfer  mentioned  concerning  the  material  measured 
by  Items  4,  S,  6  and  7. 

In  the  final  area.  Evaluation,  a  different  trend  is  evident.  Most  of  the  students  have 
a  significant  amount  of  misinformation  concerning  the  Characteristics  of  Evaluation 
(Item  20),  and  Test  Construction  (Items  21  and  22).  As  with  previously  mentioned 
areas,  considerable  alteration  of  strategy  must  be  made.  Many  students  are  unin¬ 
formed  about  Methods  of  Grading,  Item  Analysis,  and  Performance  Rating  areas 
(Items  23,  24  and  2S).  Nearly  two-thirds  (62%)  indicated  that  they  are  uninformed 
about  Grading,  75%  on  Item  Analysis,  and  27%  on  Performance  Ratings.  Once  again, 
this  information  would  require  an  alteration  of  the  approach  used  by  the  in¬ 
structor.  However,  this  approach  would  be  different  than  the  one  used  for  students 
who  have  strong  misconceptions  concerning  the  material.  Experienced  instructors 


find  it  much  easier  to  teach  new  material  to  students  who  have  no  misconceptions 
than  to  those  who  do.  In  this  instance,  negative  transfer  is  noi  likely  to  hinder  the 
students’  learning. 

It  may  be  of  interest  to  the  reader  to  know  that  Item  23,  when  administered  as 
a  choice  test  item,  had  an  Ease  Index  of  62%.  That  is,  62%  of  the  students  taking  the 
test  responded  correctly.  This  would  indicate  that  nearly  two-thirds  of  the  students 
were  able  to  compute  a  T-Score  prior  to  instruction  when  in  fact,  only  19  students 
(23%)  had  enough  knowledge  of  the  subject  to  respond  accordingly.  This  is  a  prime 
example  of  how  misleading  choice  data  can  be  when  compared  to  how  much  the 
students  actually  know. 

Appendix  E  portrays  the  same  type  data  presented  in  the  previous  table.  It  can  be 
noted  that  some  of  the  same  items  that  were  on  the  pretest  also  appear  on  the  post 
test.  These  are  noted  by  the  asterisks  with  the  pretest  item  number  in  parentheses. 
Rather  than  looking  at  each  individual  item  on  the  test,  it  is  sufficient  to  say  that 
there  were  significant  increases  in  the  percentages  of  students  who  were  informed 
or  higher  and  a  marked  decrease  in  the  percentage  of  students  who  were  either  un¬ 
informed  or  misinformed  on  the  pretest.  For  example,  Item  24  on  the  pretest  had  a 
change  from  4%  well  informed  to  92%  on  the  post  test;  from  75%  uninformed  to 
4%,  and  from  14%  misinformed  to  9%.  This  provides  the  instructor  or  course  di¬ 
rector  more  information  concerning  the  initial  level  of  proficiency  of  the  student 
and  a  more  accurate  and  precise  measure  of  the  learning  that  has  occurred  as  a  re¬ 
sult  of  instruction  received  in  a  course.  Appendices  F  and  G  provide  similar  in¬ 
formation  for  Class  69-B. 

Although  each  individual  item  could  be  described  in  detail  for  each  test,  for  each 
class,  it  hardly  seems  necessary.  The  data  clearly  indicate  that  significantly  more 
information  is  provided  by  confidence  tests  than  the  standard  choice  tests.  This 
material  would  be  of  great  value  to  the  individual  or  committee  who  has  the  respon¬ 
sibility  for  planning  a  multi-tracked  curriculum  to  better  meet  student  needs.  This 
mode  of  testing  also  provides  a  greater  amount  and  quality  of  information  about 
the  student  than  does  choice  testing.  Thus,  the  instructor  should  be  better  able  to 
adjust  his  teaching  strategy  to  meet  the  different  situations  he  faces. 

Item  Analysis  Data 

Hypothesis:  Confidence  tests  will  yield  more  data  that  can  be  used  in  the  item 
analysis  than  will  choice  tests. 

For  any  test  to  be  consistently  effective,  care  has  to  be  exercised  *n  the  selection  of 
items  that  will  be  included  in  that  test.  Periodically,  subject  maiter  will  change,  re¬ 
quiring  the  evaluator  to  update  his  test  instrument.  To  do  this,  he  must  either  con¬ 
struct  new  items  to  measure  the  new  objectives,  or  rewrite  the  existing  items  that 


I 


are  Hill  usable.  Selecting  items  to  be  included  on  a  lest  is  not  a  task  that  is  accom¬ 
plished  intuitively,  but  rather  is  a  systematic  procedure  that  requires  careful  con¬ 
sideration  be  given  to  the  item  analysis  data  available.  II'  the  evaluator  has  such  in¬ 
formation  as  an  Ease  Index.  Differentiation  Index,  and  response  distribution  data 
at  his  disposal,  the  task  will  be  greatly  simplified  and  he  is  more  likely  to  select 
items  that  will  effectively  measure  the  objectives  reliably  and  with  a  high  degree  of 
validity.  The  above  mentioned  data  can  easily  be  obtained  from  the  choice  test. 
However,  there  is  reason  to  believe  that  confidence  tests  will  not  only  provide  the 
same  data  but  also  additional  information,  perhaps  with  a  greater  degree  of 
accuracy. 

This  study  found  that  the  basic  formula  for  computing  the  base  Index5  could  be 
used  in  the  confidence  testing  situation  by  using  the  number  of  students  who  were 
classified  as  Well  Informed  and  Informed  as  a  substitute  for  the  letter  R  in  the  for¬ 
mula.  That  is,  these  are  the  students  who  would  have  answered  the  item  correctly 
on  the  choice  test  since  they  placed  either  all  of  their  confidence  or  at  least  most  of 
it  on  the  keyed  response.  For  the  letter  N.  the  total  number  of  students  taking  the 
test  was  substituted.6  From  this  data,  the  formula  could  be  executed  in  a  normal 
manner.  It  was  found  that  the  indices  obtained  from  this  method  were  approxi¬ 
mately  the  same  as  those  that  were  computed  for  the  same  items  taken  as  a  choice 
test,  with  minor  deviations  in  both  directions  for  the  guessing  involved  by  those 
students  classified  as  Uninformed  and  Partially  Informed. 


5  The  formula  used  for  computing  the  l  av:  Index  is 
El  •  —  x  100 

where  R  *  the  number  of  students  responding  correctly, 
and  N  ■  the  total  number  of  students  in  the  group. 


6  Consideration  was  given  to  omitting  the  students  who  were  classified  as  Uninformed  and  Par¬ 
tially  Informed  from  the  computation,  since  they  hud  not  made  a  commitment  to  a  single- 
answer.  However,  if  an  accurate  indication  is  to  be  made  as  to  the  Ease  of  the  item  for  the 
entire  group,  the  decision  was  made  to  use  all  students  in  the  group  when  making  this 

calculation. 


The  simple  Differentiation  Index  formula7  was  treated  in  basically  the  same  manner; 
that  is,  only  those  students  that  were  classified  as  either  Well  Inluimed  or  Informed 
were  included  The  substitution  of  values  is  listed  below.  Like  the  computation  of 
the  Lase  Index,  there  were  only  minor  deviations  from  the  Index  computed  for  the 
choice  test. 

In  spite  of  the  finding  that  there  were  minimal  differences  between  the  indices 
when  computed  for  both  choice  and  confidence  tests,  it  seems  likely  that  the  re¬ 
sults  obtained  from  the  confidence  computations  would  be  a  more  valid  assessment 
of  the  item's  performance  than  the  choice  calculations.  This  is  based  on  the 
assumption  that  the  guessing  factor  was  eliminated  from  the  confidence  indices. 
Thai  is.  the  students  who  appeared  to  be  guessing  on  the  choice  test  (those  who 
were  Uninformed  and  Partially  Informed!  were  eliminated  from  the  calculations  of 
the  Confidence  Lase  and  Differentiation  Indices. 

Additionally,  the  states  of  knowledge  of  the  upper  third  of  the  students  and  the 
lower  third  could  he  easily  plotted  on  a  graph,  thus  introducing  a  different  form 
of  displaying  the  Differentiation  Index  for  the  confidence  test.  Table  1 1  portrays 
the  graphic  representation  of  the  states  of  knowledge  of  the  high  and  low  thirds  of 
the  students  for  eight  items  on  the  pretest  for  Class  hh-A.  The  more  vertical  the 
“W"  and  “I”  lines,  the  more  effective  the  item  was  in  differentiating  among  stu¬ 
dents.  Also,  the  longer  the  “W"  line,  the  easier  the  item  was  lor  the  two  groups. 
(Note:  It  should  not  be  interpreted  as  the  relative  ease  of  the  item  for  the  whole 
group  since  the  middle  third  of  the  class  is  not  considered  when  computing  the  D.  I.. 
and  is  omitted  from  the  plots.)  Similarly,  the  more  horizontal  the  and  “M" 
lines,  the  more  confidence  placed  in  an  incorrect  answer  by  the  lower  third  of  the 
class.  The  shorter  the  “C”  and  “M”  lines,  the  less  misinformation  possessed  by 
these  two  groups.  Ideally,  there  should  be  only  minimal  length  of  the  “C"  and  “NT* 
lines  since  it  is  highly  desirable  that  all  students  successfully  learn  the  material  and 
evaluate  their  knowledge  realistically.  However,  since  all  students  will  not  be 
equally  knowledgeable,  it  is  desirable  that  the  misinformation  be  identified. 
Usually,  misinformation  is  more  likely  to  occur  with  those  students  who  achieve  the 
lower  scores  and.  in  turn,  have  mastered  less  subject  matter  than  those  with  higher 
scores. 


I'he  formula  used  toeompute  the  Differentiation  Index  was: 


1)1  = 


Hr  L 
N/3 


w  here  It  =  the  numher  of  students  in  the  high  one-third  of  the  group  who  were  elassified  as 
W  or  1.  and  I  =  those  in  the  low  one-third  who  were  classed  as  \\  or  I.  N  =  the  total 
number  of  students  in  the  group,  and  3  -  Numher  of  Criterion  Ciroups. 
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HIGH  HIGH 


Table  1 1(b).  State:  of  Knowledge  of 


ITEM  42 


0  5  10  15  20 

LOW 


Thus,  with  the  ability  to  compute  two  basic  measures  of  an  item's  effectiveness,  the 
Ease  and  Differentiation  Indices,  along  with  the  ability  to  display  the  Differen¬ 
tiation  in  states  of  knowledge  in  a  graphic  manner,  it  appears  that  the  hypothesis  is 
partially  confirmed.  However,  an  evaluator  also  uses  the  analysis  of  response  effec¬ 
tiveness  in  order  to  determine  how  effectively  the  distracting  alternatives  are  per¬ 
forming.  With  the  standard  choice  test,  this  is  done  by  plotting  the  number  of  re¬ 
sponses  for  each  item  by  alternative  and  by  criterion  group.  This  tally  for  a  typical 
item  appears  as  follows: 


Item  25  Ease  Index:  59%,  Differentiation  Index:  .48 


Alternatives  A 

High  1/3  1 

Middle  1/3  3 

Low  1/3  7 

Totals:  11 


B*  C  D 

20  2  2 

16  3  3 

8  5  5 

44  10  10 


With  this  data,  he  can  compute  the  Ease  and  Differentiation  indices  by  using  the 
numbers  plotted  for  the  high  and  low  criterion  groups  in  the  alternative  “B" 
column  along  with  the  total  number  of  students  in  the  group.  Additionally,  he  can 
observe  the  quality  of  the  distractors  by  visual  inspection.  When  he  finds  that  no 
one  has  selected  a  particular  alternative,  he  can  assume  that  it  is  not  plausible  and 
replace  it  with  one  which  is. 

However,  from  the  above  data  the  evaluator  is  unable  to  determine  if  a  student 
even  considered  selecting  an  alternative  other  than  his  actual  choice.  Confidence 
testing  can  provide  this  information,  thus  indicating  that  the  alternative  was 
actually  more  plausible  than  it  appeared  on  the  choice  test. 


Since  u  student  can  assign  some  degree  of  confidence  to  all  possible  answers  he  feels 
may  be  correct,  an  analysis  of  these  levels  of  confidence  can  be  made.  This  analysis 
will  provide  a  better  indication  of  how  plausible  the  distractor  actually  was.  Tables 
12  through  17  contain  data  that  indicates  how  many  times  each  level  of  confidence 
was  used  and  the  cumulative  frequency  for  the  various  levels.  Additionally,  the 
cumulative  frequency  has  been  plotted  graphically  for  each  of  the  items,  providing  a 
visualization  of  the  relative  effectiveness  of  each  of  the  alternatives.  For  example. 
Item  I  indicates  that  most  of  the  6K  students  in  69-A  had  a  great  deal  of  confidence 
in  the  correct  answer  (alternative  number  IK  With  alternative  2.  most  students(57) 
assigned  no  confidence  to  it  as  being  possibly  correct,  while  three  were  positive  it 
was  the  correct  answer.  Were  (his  item  considered  from  a  choice  test  standpoint, 
only  the  three  students  who  assigned  complete  co  ifidence  to  this  alternative  would 
have  appeared  on  the  tally  mentioned  previously.  There  is  a  possibility  that  one.  two 
or  all  three  of  the  students  who  assigned  half  their  confidence  (M  level)  would  have 
selected  this  as  their  answer,  depending  upon  how  they  guessed.  However,  none  of 
the  remaining  live  students  would  have  given  any  indication  that  they  had  even  con¬ 
sidered  this  as  being  possibly  correct,  since  they  had  more  confidence  in  another  al¬ 
ternative.  Thus,  we  find  (hat  the  distractor  is  working  more  effectively  than  one 
might  imagine. 

Similarly,  alternative  3  had  a  slightly  better  distracting  ability  than  did  alternative 
2.  since  two  fewer  students  were  able  to  eliminate  this  as  a  possibility.  There  were 
three  students  who  placed  some  degree  of  confidence  on  the  alternative  that  would 
not  have  been  recognized  with  choice  testing.  Likewise,  four  responses  for  alterna¬ 
tive  4  would  not  have  been  discovered. 

In  a  more  dramat.c  representation.  Item  57,  (Tables  15-17).  clearly  indicates  how 
the  test  constructor  might  be  misled  when  determining  the  effectiveness  of  an  item's 
distractors.  With  the  keyed  (correct )  response  ( I )  there  weie  nine  students  who  had 
some  confidence  in  the  response,  but  would  not  have  selected  it  by  choice.  This 
would  indicate  they  at  least  had  enough  knowledge  concerning  the  subject  matter 
to  consider  it  as  being  a  possibility.  Also,  nine  students  divided  their  confidence 
equally  between  the  keyed  response  and  another  alternative  Depending  upon  how 
successful  they  were  in  guessing,  the  base  Index  for  this  item  could  vary  by  23%. 
thus  giving  an  inaccurate  indica  <on  of  its  effectiveness  and  relative  ease.  In  alterna¬ 
tive  3. 13  students  indicated  degrees  of  confidence  that  would  not  have  been  identi¬ 
fied.  Alternative  4  had  1 1  that  would  not  have  been  identified. 


Table  12.  Number 

of  Times 

Each  Degree 

of  Confidence  Used  for 

Item  1  (post  test). 

Level  of 

Alternative 

Alternative 

Alternative 

Alternative 

Confidence 

Number  1 

Number  2 

Number  1 

Number  4 

A  (None) 

17 

57 

55 

56 

B 

1 

1 

1 

C 

1 

1 

D 

2 

1 

1 

E 

1 

F 

C 

H 

1 

I 

1 

1 

1 

J 

K 

L 

M 

3 

3 

2 

2 

N 

0 

P 

1 

Q 

R 

2 

S 

T 

U 

1 

V 

1 

W 

1 

X 

Y 

Z  (Complete) 

43 

3 

6 

5 

TOTAL 


68 


68 


68 


68 
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Table  13.  Frequency  Distribution  of  Item  1. 


Level  of 
Confidence 


Alternative  Alternative  Alternative  Alternative 
Number  I  Number  2  Number  3  Number  4 


Z  (Complete) 

Y 
X 
W 

V 

u 

T 

s 

R 

Q 

P 

0 

N 

M 

L 

K 

J 

I 


(None) 


43  3 


6  5 


44 


7 

8 


46 

6 

8 

49  6  10 


50  119 

7 

12 

9  13  10 

10  n 

51  11  12 

68  68  68  68 


Table  14(a).  Total  Number  of  Students  (Class  hOA)  Who  Expressed  at 
Least  Some  Confidence  in  Each  Response  to  Item  I  (post  test). 


DEGREE  OF  CONFIDENCE 


NUMBER  OF  STUDENTS  NUMBER  OF  STUDENTS 


Table  14{b).  Total  Number  of  Students  (Class  69A)  Who  Expressed  at 
Least  Some  Confidence  in  Each  Response  to  Item  I  (post  test). 
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Table  15.  Number  of  Times  Each  Degree  of  Confidence 
Used  for  Item  57  (post  test). 


Level  of 
Confidence 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

0 

P 

Q 

R 

S 

T 

U 

V 

w 

X 

Y 
Z 

TOTAL 


Alternative 

Alternative 

Alternative 

Alternative 

Number  1 

Number  2 

Number  3 

Number  4 

12 

54 

46 

51 

1 

2 

1 

1 

2 

2 

1 

1 

2 

3 

3 

3 

3 

4 

2 

1 

2 

1 

1 

1 

1 

1 

9 

2 

4 

3 

1 

1 

2 

1 

1 

1 

1 

32 

6 

4 

2 

68 

68 

68 

68 

Table  16.  Cumulative  Frequency  Distribution  for  Item  57. 


Level  of 

Alternative 

Alternative 

Alternative 

Alternative 

Confidence 

Number  1 

Number  2 

Number  3 

Number  4 

Z 

32 

6 

4 

2 

Y 

33 
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w 

34 

V 

3 

u 

5 

T 

36 

S 

R 

37 

Q 

38 

p 

0 

N 

M 

47 

8 

9 

6 

L 

K 

48 

10 

J 

49 

11 

1 

51 

9 

13 

7 

H 

G 

54 

12 

16 

11 

F 

E 

55 

18 

14 

D 

56 

C 

13 

20 

16 

B 

14 

22 

17 

A 

68 

68 

68 

68 

NUMBER  OF  STUDENTS  NUMBER  OF  STUDENTS 


Tabic  17(a).  Total  Number  of  Students  (Class  o‘>A)  Who  Impressed  at 
Least  Some  Confidence  in  Bach  Response  to  Item  57  (post  test). 


Alternative  1 


0  .2  .4  .6  .8  1.0 


DEGREE  OF  CONFIDENCE 


NUMBER  OF  STUDENTS  NUMBER  OF  STUDENTS 


Table  17(b).  Total  Number  of  Students  (Class  69A)  Who  Expressed  at 
Least  Some  Confidence  in  Each  Response  to  Item  57  (post  test). 


Alternative  3 


DEGREE  OF  CONFIDENCE 


Still  another  visual  representation  can  be  made  of  the  data  provided  by  a  confidence 
test.  This  representation  involves  the  same  basic  technique  mentioned  above,  that  is, 
using  the  cumulative  frequencies  of  the  levels  of  assigned  confidence,  except  that 
the  high  third  of  the  group  and  the  low  third  have  been  combined  on  each  graph. 
This  provides  another  means  of  observing  the  differences  in  the  responses  of  the 
better  students  and  the  weaker  ones.  Computing  the  Differentiation  Index,  as  men¬ 
tioned  earlier,  provides  a  numerical  indication  of  how  effectively  the  items  differen¬ 
tiate  among  students  in  designed  criterion  groups.  The  charts  contained  inTahles  18 
and  19  provide  a  visualization  of  that  information  and  also  display  the  differences 
in  levels  of  confidence  assigned  to  each  of  the  distractors  by  each  of  the  groups.  For 
example.  Item  I  on  Table  18  shows  that  there  were  substantial  differences  in  the 
levels  of  confidence  assigned  to  the  correct  answer  (alternative  I ).  Alternative  2 
indicates  that  more  of  the  weaker  students  placed  more  confidence  in  this  answer 
than  did  the  better  students.  For  the  lower  one-third,  it  sounded  more  plausible  than 
it  was  for  the  upper  third.  Alternative  3  discloses  that  approximately  the  same  num¬ 
bers  of  top  and  bottom  students  placed  about  the  same  degrees  of  confidence  in  the 
answer.  The  last  alternative  for  this  item  clearly  shows  that  the  lower  third  found 
this  choice  more  plausible  than  did  students  in  the  top  third.  The  reader  will  note 
that  no  student  in  the  top  third  place  more  than  5 O'*'  of  his  confidence  in  that 
response,  while  four  students  in  the  lower  third  placed  complete  confidence  in  it  as 
being  the  correct  answer.  With  this  manner  of  representing  the  differences  in  as¬ 
signed  confidence  levels,  the  greater  the  area  between  the  upper  levels  of  the  lines,  the 
greater  the  differentiation  between  groups.  With  this  form  of  data  reporting,  it  is 
possible  to  compute  some  form  of  “differentiation  index”  among  the  alternatives. 

Ideally,  there  should  be  a  significantly  large  area  between  the  assigned  levels  of  con¬ 
fidence  for  the  top  and  bottom  groups  on  the  correct  answer,  with  the  top  group 
greater  in  number  than  that  of  the  lower.  With  the  distractors.  the  lower  group’s 
level  of  confidence  line  should  exceed  the  top  group's,  since  the  lower  group  has 
likely  failed  to  master  the  subject  matter  and  would  most  likely  display  greater  con¬ 
fidence  in  an  incorrect  alternative  than  would  the  students  who  were  placed  in  the 
top  group  based  upon  their  overall  test  score.  Item  25  on  Table  18  illustrates  this 
point.  Alternative  3  appears  to  be  an  excellent  distractor.  since  it  caused  a  sufficient 
number  of  lower  students  to  place  complete  confidence  in  it.  The  keyed  response, 
alternative  4,  clearly  indicates  that  most  of  the  students  in  the  top  third  assigned  a 
high  degree  of  confidence  to  this  answer,  while  approximately  one-half  of  the  lower 
third  assigned  corresponding  levels  of  confidence  to  it. 

Each  item  can  be  covered  in  detail  by  the  reader,  but  it  is  evident  that  this  method 
of  analyzing  the  effectiveness  of  the  alternatives  provides  moie  information  than 
can  be  obtained  with  a  normal  choice  test.  The  above  information,  along  with  the 
ability  to  compute  the  Ease  and  Differentiation  Indices  with  more  validity  and  to 
display  the  findings  in  other  than  numerical  terms,  tends  to  confirm  the  hypothesis 
of  :!ii>  p.o;  of  the  ut.  Jy. 
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CONCLUSION 


Summary  of  Major  Relationships 

The  problems  stated  at  the  outset  were:  to  discover  whether  information  provided 
by  confidence  testing  would  result  in  a  more  accurate  assessment  of  student  knowl¬ 
edge  so  more  personal  individualized  instruction  could  be  presented,  and  whether 
sufficient  precise  data  could  be  attained  to  be  of  significant  aid  in  curriculum 
planning.  Analysis  has  shown  that  much  qualitative  and  quantitative  information 
is  provided  when  employing  the  methods  and  materials  associated  with  confidence 
testing.  A  compendium  of  those  findings  follows: 

Predictive  Validity  appears  to  be  positive,  but  not  significantly  high  enough  to 
forecast  a  student's  performance  on  practice  teaching  lessons  01  Ins  final  standing 
in  the  class.  This  is  more  strongly  associated  with  the  officer  class  Ilian  with  the  en¬ 
listed  class.  Choice  pretest  scores  have  a  slightly  higher  predictive  value  than  do  con¬ 
fidence  scores  for  the  enlisted  class.  Neither  choice  nor  confidence  pretest  scores 
correlate  significantly  enough  with  success  at  AIC  to  warrant  use  as  predictive 
devices. 

The  Assessment  of  Student  Knowledge  tends  to  be  more  accurate  when  using  confi¬ 
dence  tests.  Data  presented  in  this  section  of  the  report  clearly  indicate  that  most 
students  would  have  been  misclassified  as  to  their  actual  levels  of  knowledge  if  only 
choice  test  results  had  been  used.  The  frequency  of  reversals  of  tank  order  position 
indicate  that  when  a  student's  actual  state  of  knowledge  is  more  accurately  assessed 
his  position  within  the  group  changes,  thus  reflecting  more  accurate  differentiation. 
There  are  strong  reasons  to  believe  that  students  tend  to  evaluate  their  knowledge 
more  carefully  and  critically  when  using  confidence  tests,  and  the  more  often  they 
are  tested  with  the  materials,  the  more  realistic  this  evaluati  m  becomes. 

Reliability  of  test  instruments  seems  to  be  improved  when  using  confidence  tests. 
In  the  one  instance  when  the  actual  reliability  coefficient  was  computed,  a  signifi¬ 
cant  increase  was  noted.  This,  coupled  with  the  fact  that  significant  numbers  of 
guessing  situations  were  eliminated  by  using  the  confidence  test,  indicates  that  the 
accuracy  and  reliability  of  the  instrument  is  likely  to  be  increased. 

Confidence  tests  provide  more  Information  about  the  students  than  do  choice  tests. 
The  ability  of  the  confidence  test  to  categorize  states  of  knowledge  lor  each  item  on 
a  pretest  would  make  multi-tracking  decisions  easier  for  curriculum  planners.  The 
ability  of  the  confidence  test  to  identify  the  various  levels  of  pioticieiicy  for  each 
student  in  each  area  of  the  curriculum  provides  an  invaluable  aid  to  the  instructor 
in  planning  his  teaching  strategy.  By  being  able  to  identify  students  who  need 


supplementary  material,  who  need  reinforcement,  who  are  apt  to  have  difficulties 
because  of  negative  transfer  of  learning  and  who  know  nothing  jhout  the  material 
to  be  presented,  the  instructor  can  more  readily  adapt  to  meet  the  needs  of  each 
individual. 

Better  Item  Analysis  Data  is  provided  with  confidence  tests  than  with  the  standard 
choice  tests.  Both  methods  of  testing  provide  the  data  to  compute  the  basic 
measures  of  an  item’s  effectiveness,  but  confidence  tests  provide  a  more  accurate 
index  since  responses  involving  guessing  are  not  a  factor  in  the  computations.  The 
ability  of  the  confidence  test  to  identify  those  students  who  cannot  rule  out  the 
incorrect  responses  to  an  item  provides  valuable  data  to  the  individual  constructing 
or  analyzing  the  test.  A  more  accurate  assessment  can  he  made  of  the  detractor's 
effectiveness. 

Suggestions  for  Further  Research 

The  analysis  reported  here  is  basic  and  exploratory  in  nature  and.  although  it  does 
test  several  specific  hypotheses,  it  needs  more  statistical  refinement.  Modifications 
in  design,  statistical  analysis,  and  inclusion  of  new  variables  should  be  considered 
for  further  research  relating  to  the  use  of  confidence  testing  in  military  situations. 
The  following  areas  are  suggested  for  future  research: 

1.  The  use  of  the  confidence  testing  materials  and  techniques  with  the  Allied 
Officers  who  attend  the  Allied  Officer  Familiarization  Course  (AGIO.  This  area 
appears  to  be  one  in  which  these  techniques  could  be  especially  useful  with  those 
students  who  are  having  some  difficulty  with  the  written  portion  of  the  Fnglish 

language. 

2.  Continued  research  with  different  variables  to  determine  if  the  technique  can  be 
used  to  measure  all  aspects  of  affective  as  well  as  cognitive  learning  areas.  The 
ability  of  a  written  test  instrument  to  measure  such  things  as  altitude,  interests,  and 
appreciations  has  not  been  successfully  developed,  thus  opening  the  possibility  for 
additional  research. 

3.  Continued  research  with  confidence  testing  to  determine  the  amount  of  repeti¬ 
tion  that  is  needed  if  the  test's  validity  and  reliability  is  to  be  maintained.  Theo¬ 
retically,  if  all  items  were  sound  qualitatively  repetition  would  be  eliminated  without 
decreasing  the  reliability  of  the  instrument. 

4.  Research  oriented  toward  the  use  of  confidence  tests  m  perloi  mancc  situations. 
The  elements  of  exercises  such  as  practice  teaching  lessons  could  be  tested  prior  to 
the  student  performance  to  determine  if  the  written  lest  can  predict  success  in 
actual  performance. 
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Appendix  A:  Summary  of  the  Relationships  between  V; 


CM  on  005  co  <-  in  cm  ®  '  00  «-  o>  co  oot^  cm  o  n  n  v  «-  oo 

t—  r^M;  p  — _  r<:  M;  ♦  v  o  cm  <r  p  P  p  cm  cm  in  o>  oo  p  p 

*-  os  co  ©  cm  in  <o  cm  co  com-  n  <o  r»  «-  con  in--  p  ■-  pp 

r-  2}  p  fi  cm  c*  ^  «*  "SF  cm  ^  n  cm  co  p  «-  p  in  in  So 

O  in  oo  0>©  cm  cm  in  a  r*o  *-  cm  r>«  c>  <o  oi  oo  oo 

t-  m;  p  PP  sf  p  N  n  iq  p  to  p  <q  p  oo  p  r-  p  oo 

®  p  S  8S  2  8  2  8  S[8  5  2  3  2  2?  88 

r-“  f— 

m  C!  r:  Qt  2  ^  ♦  m  *  ®  in  sr  m  —  oo 

w  p  p  p  o  p  cm  Cm  cm  p  c*  r>.  in.  co  oo 

r»  r* 

*  88  88  58  n  3  88  88 


|®S888SS2  5  8^88 

g  •“  - 

•5 

«  82  28  82  88 

*— 

«  88  88  8$  88 


«  5R  2  2  8  8 
m  88  88 


c 

15 

«*■ 

c 

<n 

0 

Z 

I 

*0 

8 

♦* 

£ 

X 

V 

0 

9 

3 

8 

h- 

K 

0. 

<5 

Jf 

4) 

O 

1 

0 

Ui 

O 

3 

5 

CL 

O 

C 

*» 

1 

i 

f 

•n 

t 

1 

0> 

l» 

3 

» 

> 

< 

K 

Q. 

to 

CC 

£ 

t- 

5 

UL 

** 

I 

(E 

h* 

CL 

(3 

8 

-J 

*<5 

.  c 

II 

I 

il 

1 

CN 

cn 

ir> 

(b 

rs! 

00 

d 

d 

v' 

12.  Post  Test  (Confidence) 


Appendix  B:  Post  Test  Results  (Class  69A) 


Student 

Post  Test  Score 

Rank  Order 

Number 

Confidence 

Choice 

Confidence 

Choice 

1  -  2 

72.30 

71 

1 

1.5 

0-  2 

71.71 

71 

2 

1.5 

K-  4 

70.69 

69 

3 

6.0 

G-  5 

70.44 

69 

4 

6.0 

N-  4 

70.00 

70 

5 

3.5 

H-  7 

69.03 

67 

6 

11.0 

N-  3 

69.00 

70 

7 

3.5 

I  •  1 

68.90 

63 

8 

25.0 

G-  4 

68.89 

69 

9 

6.0 

L-  3 

68.81 

67 

10 

11.0 

J  -  6 

68.66 

68 

11 

8.5 

J  -  7 

68.06 

64 

12 

20.5 

G-  3 

67.87 

65 

13 

16.5 

J  •  2 

67.46 

68 

14 

8.5 

F  -  1 

66.89 

67 

15 

11.0 

I  -  5 

66.83 

62 

16 

30.0 

L  •  4 

66.77 

65 

17 

16.5 

I  -  4 

66.74 

63 

18 

25.0 

N*  2 

66.25 

64 

19 

20.5 

K  •  2 

66.00 

66 

20.5 

13.5 

N-  5 

66.00 

66 

20,5 

13.5 

0-  7 

65.89 

61 

22 

33.0 

G  -  6 

65.88 

65 

23 

16.5 

J  -  3 

65.70 

63 

24 

25.0 

G-  1 

65.68 

62 

25 

30.0 

L  -  6 

64.84 

65 

26 

16.5 

L  -  5 

64.68 

64 

27 

20.5 

N-  7 

64.52 

62 

28 

30.0 

0-  5 

64.42 

64 

29 

20.5 

F  •  2 

64.07 

62 

30 

30.0 

B  -  4 

64.00 

63 

31 

25.0 

B  •  3 

63.86 

63 

32 

25.0 

0  -  6 

63.56 

59 

33 

37.0 

L-  2 

62.49 

59 

34  ’ 

37.0 

Appendix  B: 

Post  Test  Results  (Class  69A) 

Student 

Post  Test 

Score 

Rank 

Order 

Number 

Confidence 

Choice 

Confidence 

Choice 

0- 

1 

62.31 

57 

35 

44  0 

1 

7 

62.23 

57 

36 

44.0 

i 

4 

62.07 

62 

37 

30.0 

H 

5 

62.00 

54 

38 

61.5 

K 

6 

61.94 

54 

39 

61,5 

H 

4 

61.84 

60 

40 

14.0 

L 

7 

61.37 

56 

41 

46.0 

F 

5 

60.89 

55 

42 

48.5 

H 

1 

60.73 

59 

43 

37.0 

J 

1 

60.54 

59 

44 

37.0 

J 

5 

6'  30 

51 

45 

53.0 

G 

7 

60.00 

59 

46 

37.0 

I 

6 

59.94 

49 

47 

59.0 

H 

6 

59.90 

55 

■48 

48.5 

0 

3 

59.42 

55 

49 

48.5 

I 

3 

59.05 

52 

50 

53.0 

H 

2 

59.00 

58 

51 

41.0 

K 

5 

58.44 

58 

52 

41.0 

B 

6 

58.37 

58 

53 

41.0 

H 

3 

58.10 

57 

54 

44.0 

B 

7 

58.08 

55 

55 

48.5 

K 

2 

57.26 

50 

56 

57.5 

B 

1 

56.93 

51 

57 

55.0 

F 

6 

55.36 

44 

58 

63.5 

B 

2 

53.95 

61 

59 

55.0 

G 

2 

53.69 

50 

60 

57.5 

L 

1 

52.59 

48 

61 

60.5 

K 

3 

50.73 

34 

62 

67.0 

N 

6 

49.98 

45 

63 

62.0 

G 

4 

49.36 

44 

64 

63.5 

N 

1 

48.52 

48 

65 

60.5 

F 

3 

42.03 

39 

66 

65.0 

0 

4 

37.42 

31 

67 

68.0 

B 

5 

37.00 

37 

68  • 

66.0 

Appendix  C:  Post  Test  Results  (Class  69B) 


Student 

Post  Test  Score 

Rank  Order 

Number 

Confidence 

Choice 

Confidence 

Choice 

H-  5 

45.84 

44 

1 

1.5 

E  -  6 

45.37 

43 

2 

4.0 

E  -  8 

44.43 

44 

3 

1.5 

H-  4 

44.16 

41 

4 

10.0 

1  -  5 

44.03 

43 

5 

4.0 

J  -  2 

43.87 

40 

6 

14.0 

I  -  3 

43.21 

40 

7 

14.0 

P-  5 

42.84 

42 

8 

7.0 

1  -  2 

42.73 

39 

9 

18.5 

Q-  5 

42.00 

42 

10 

7.0 

E-  3 

41.96 

39 

11 

18.5 

L-  2 

41.84 

41 

12 

10.0 

Q-  7 

41.69 

42 

13 

7.0 

L-  8 

41.64 

41 

14 

10.0 

I  -  6 

40.87 

35 

15 

30.0 

Q-  6 

40.39 

40 

16 

14.0 

H-  6 

40.20 

40 

17 

14.0 

G-  3 

40.00 

40 

18 

14.0 

M-  2 

39.29 

39 

19 

18.5 

M-  1 

39.00 

39 

20 

18.5 

E  -  2 

38.58 

32 

21 

48.5 

T  ■  5 

38.39 

29 

22 

60.5 

H-  2 

38.33 

29 

23 

60.5 

G-  6 

38.27 

35 

24 

30.0 

E-  5 

38.20 

37 

25 

24.5 

N-  6 

38.18 

36 

26 

26.5 

I  -  8 

38.02 

33 

27 

41.5 

M-  3 

38.00 

38 

28  ■ 

22.0 

Appendix  C:  Post  Test  Results  (Class  69B) 


Student 

Post  Test  Score 

Rank  Order 

Number 

Confidence 

Choice 

Confidence 

Choice 

J  -  3 

37.58 

38 

29 

22.0 

E-  1 

37.42 

36 

30 

26.5 

J  •  4 

37.35 

33 

31 

41.5 

Q-  1 

37.20 

38 

32.5 

22.0 

Q-  3 

37.20 

37 

32.5 

24.5 

G-  4 

36.79 

34 

34 

35.0 

Q-  2 

36.74 

34 

35 

35.0 

L-  1 

36.62 

35 

36 

30.0 

E-  7 

36.58 

33 

37 

24.5 

M-  5 

36.52 

32 

38 

48.5 

T-  2 

36.41 

34 

39 

35.0 

H-  8 

36.20 

43 

40 

4.0 

G-  8 

35.79 

30 

41 

56.0 

J  •  1 

35.39 

34 

42 

35.0 

T-  6 

35.37 

30 

43 

56.0 

N-  1 

35.18 

25 

44 

73.5 

L-  3 

35.14 

32 

45 

48.5 

H-  1 

34.67 

32 

46 

48.5 

P-  1 

34.52 

33 

47 

41.5 

M-  6 

34.24 

32 

48 

48.5 

H-  7 

34.00 

33 

49.5 

41.5 

P-  4 

34.00 

34 

49.5 

35.0 

T-  3 

33.90 

31 

51 

52.5 

N-  4 

33.84 

35 

52 

30.0 

T-  1 

33,76 

17 

53 

83.0 

M-  4 

33.57 

35 

54 

30.0 

L-  6 

33.22 

33 

55 

41.5 

J  -  8 

33.15 

25 

56 

73.5 

Appendix  C:  Post  Test  Results  (Class  69B) 


Student 

Post  Test  Score 

Rank  Order 

Number 

Confidence 

Choice 

Confidence 

Choice 

J  -  5 

33.02 

31 

57.5 

52.5 

J  -  6 

33.02 

24 

57.5 

78.0 

T  •  7 

32.77 

27 

59 

67.5 

J  ■  7 

32.74 

27 

60 

67.5 

P-  2 

32.68 

33 

61 

41.5 

1  •  7 

32.66 

16 

62 

84.0 

P-  6 

32.59 

30 

63 

56.0 

N-  5 

32.49 

28 

64 

64.0 

Q-  4 

32.00 

32 

65 

48.5 

M-  7 

31.84 

30 

66 

56.0 

I  •  1 

31.31 

20 

67 

82.0 

1  -  4 

30.59 

29 

68 

60.5 

L-  5 

30.56 

28 

69 

64.0 

G-  7 

30.48 

25 

70 

73.5 

P-  7 

30.00 

30 

71 

56.0 

N-  3 

29.76 

24 

72 

78.0 

T-  8 

29.64 

28 

73 

64.0 

H-  3 

29.44 

27 

74 

67.5 

G-  5 

29.00 

29 

75 

60.5 

N-  7 

28.74 

24 

76 

78.0 

N-  2 

28.00 

25 

77 

73.5 

G-  1 

27.99 

27 

78 

67.5 

N-  8 

27.94 

21 

79 

81.0 

P-  3 

27.00 

33 

80 

41.5 

L-  7 

26.75 

25 

81 

73.5 

P-  8 

25.88 

26 

82 

70.0 

L-  4 

23.76 

25 

83 

73.5 

G-  2 

22.75 

22 

84 

80.0 
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